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Abstract 

Test  and  Evaluation  (T&E)  of  the  National 
Missile  Defense  (NMD)  system  is  an  expensive  and 
complex  undertaking.  A  challenge  facing  the  NMD 
T&E  community  is  allocation  of  scarce  test  resources  to 
achieve  necessary  system  testing  within  available 
funding.  This  allocation  is  done  on  the  basis  of 
perceived  value  across  a  number  of  test  methodologies 
of  which  Integrated  Ground  Tests  (IGTs)  are  one.  This 
paper  is  based  on  identified  value  and  lessons  learned 
during  recent  NMD  integrated  ground  testing.  The 
primary  objective  of  this  paper  is  to  identify  and 
explore  the  value  of  IGTs  by  showing  how  they  meet 
NMD  T&E  challenges.  These  challenges  are  associated 
with  cost,  fidelity  and  completeness  of  system 
representation,  and  utility.  The  IGT  detailed  design  and 
implementation  process  is  discussed  in  relation  to 
interactions  with  the  NMD  T&E  community  and  the 
integrated  system  test  tool  developer.  A  description  of 
the  process  is  provided.  Recent  lessons  learned  are 
presented  with  insight  into  how  to  apply  them  to 
increase  the  value  of  future  IGTs.  Unique  IGT  value  is 
identified.  It  is  also  shown  that  IGT  value  is  synergistic 
with  the  value  of  other  test  methods  and  processes. 

Introduction 

This  paper  is  organized  in  three  parts.  Part  I 
presents  an  overview  of  the  challenge  involved  in 
ground  testing  of  an  NMD  system  and  derines  both  a 
generic  IGT  process  and  the  value  expected  to  be 
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derived  from  this  process.  Part  11  describes  the  IGT 
detailed  design  and  implementation  process,  identifies 
more  detailed  and  recently  identified  IGT  value,  and 
introduces  flexible  variations  of  the  process.  Part  III 
explores  recent  IGT  lessons  learned  with  focus  on 
increasing  the  value  of  future  ground  testing. 

I.  Overview 

Known  Challenges 

It  has  been  observed  that  test  is  the  conscience  of 
the  development  and  acquisition  process.  The 
fundamental  purpose  of  test  and  evaluation  in  this 
process  is  to  identify  risk  areas  to  be  reduced  or 
eliminated.*  A  recent  independent  DoD  study 
concluded  that  Ballistic  Missile  Defense  (BMD) 
programs  need  to  pay  more  attention  to  ground  testing, 
simulation,  and  analyses  so  as  to  reduce  issues  to  be 
resolved  in  flight  tests  to  those  that  can’t  be  resolved 
with  ground  testing  or  simulations.^  It  also 
recommended  a  formal  process  to  ensure  full 
certification  of  the  system  before  each  flight  test.  Of 
course,  not  all  data  required  for  making  a  system 
deployment  decision  is  available  from  flight  tests.  A 
considerable  body  of  additional  data  is  required  to 
support  the  Deployment  Readiness  Review  (DRR)  in 
the  year  2000.  This  data  would  come  from  a  number  of 
sources  including  IGT  and  Model  and  Simulation.  So 
the  challenge  for  NMD  T&E  is  to  optimize  the  value  of 
testing  done  with  each  available  test  methodology.  This 
optimization  must  be  done  with  consideration  of  real 
world  constraints  that  limit  the  value  derived  from  each 
methodology. 

Consider  the  major  available  system  test 
methodologies  for  the  NMD  system.  These  are 
Integrated  Flight  Test  (IFT),  Integrated  Ground  Test, 
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and  Model  and  Simulation  (M&S).  An  IFT  involves 
test  of  an  actual  representation  of  the  NMD  system  in  a 
real-world  environment.  As  a  minimum. 
Intercontinental  Ballistic  Missile  (ICBM)  target  and 
NMD  interceptor  launch(es)  at  National  Ranges  are 
involved.  These  are  very  complex  and  cosdy  at  tens  of 
millions  of  dollars  per  test.  Only  a  few  of  these  can  be 
afforded.  IFTs  are  not  suitable  for  statistical  data 
collection  due  to  test  sample  size.  However,  they  have 
the  most  credibility  since  they  come  closest  to  placing  a 
real  world  NMD  system  in  a  real  world  warfighting 
environment.  They  are  destructive  tests  that  produce 
only  limited  amounts  of  high  quality  test  data.  Risk 
reduction  for  IFTs  is  thus  imperative.  Major  IFT 
features  are  shown  in  Figure  1. 
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Figure  1.  Major  IFT  Features 

An  IGT  is  a  non-destructive  test  involving  two  or 
more  NMD  system  element  hardware/software  test 
articles  operating  in  a  system  context  in  a  conunon  test 
environment.  These  element  test  articles  consist  of 
actual  operational  software  operating  in  actual  data 
processing  hardware  of  the  element  An  example  would 
be  an  interceptor  data  processor  running  the  same 
software  it  would  in  an  actual  interceptor.  This  test 
article  is  realized  as  an  element  node  that  is  interfaced 
by  tactical  NMD  communications  to  the  other 
participating  element  test  articles.  This  element  node  is 
sometimes  referred  to  as  a  Processor  Test  Environment 
(PTE)  that  is  interfaced  to  a  Node  Test  Environment 
(NTE)  of  the  system  test  tool.  The  common  test 
environment  at  the  NTE  provides  simulated  detailed 


threat  object  dynamics,  natural  environments  (rain, 
clouds),  backgrounds  (sun,  moon,  stars,  earth, 
satellites),  and  nuclear  effects.  Only  a  minimal  amount 
of  NTE/PTE  element  interfacing  simulation  is  needed 
to  complete  the  system  representation.  The  result  is  a 
real  time  test  configuration  of  real  world  interfaced 
system  hardware/software  and  minimal  interfacing 
simulation  that  responds  as  a  system  to  a  simulated  test 
environment.  Validation  is  thus  only  required  for  the 
non-real  world  simulated  test  environment  and  the 
interfacing  simulations.  IGT  is  the  test  methodology 
next  highest  in  actual  system  realism  to  an  IFT.  IGT  is 
also  suitable  for  limited  statistical  data  collection  due  to 
the  capability  to  perform  a  moderate  number  of 
repetitive  runs.  Major  IGT  features  are  shown  in 
Figure  2. 
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Figure  2.  Major  IGT  Features 

M&S  substitutes  computer  representations  for  the 
actual  hardware  of  the  integrated  system  elements.  This 
methodology  requires  significant  accreditation  activity 
before  it  is  trusted  since  all  system  component  model 
representations  need  to  be  accredited  in  addition  to  the 
simulated  environment.  M&S  can  be  run  faster  or 
slower  than  real  time  depending  on  model  complexity 
and  executing  digital  computer  capability.  The  faster 
than  real  time  feature  makes  M&S  suitable  for  many 
repetitive  runs  and  collection  of  statistical  measures  of 
performance.  M&S  features  are  shown  in  Figure  3. 
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IGT  Process 

The  IGT  process  can  be  viewed  as  a  closed  loop 
feedback  process  that  takes  input  from  the  NMD  T&E 
community  and  the  system  elements  and  produces 
analyzed  data  and  lessons  learned  as  outputs.  Figure  4 
shows  the  major  components  of  this  process  as  1) 
system  test  tool  development  with  associated  system 
element  integration,  2)  detailed  test  design,  3)  test 
implementation,  and  4)  data  analysis. 
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builds  for  each  specific  test.  This  development  is 
intimately  associate  with  integration  of  the  evolving 
NMD  element  hardware/software  at  their  current  states 
of  maturity.  NMD  elements  are  interfaced  together  as  a 
functioning  system  representation  that  responds  to  the 
simulated  real  time  test  environment.  The  system  test 
tool  providing  the  common  test  environment  is  the 
Integrated  System  Test  Capability  (ISTC)  under 
development  in  Huntsville,  Alabama.  ISTC  was  the 
system  test  tool  used  in  the  recent  IGT  lA  test.  The 
system  test  article  representation  includes  the  necessary 
integrated  sensors  and  the  Battle  Management/ 
Command  Control  and  Communications  (BMC3)  as 
well  as  the  interceptor(s).  The  integrated  system  test 
tool  and  system  test  articles  combined  comprise  the  test 
configuration.  Each  build  of  the  test  configuration  must 
pass  incremental  integration  testing  along  with  any 
required  accreditation  or  other  testing.  Upon 
completion  of  the  build,  a  usable  system  test 
configuration  exists  for  test  implementation  at  which 
point  test  assets  are  declared  available.  This  Test  Assets 
Available  (TAA)  milestone  is  time  phased  to  occur  just 
before  the  Test  Design  Review  (TDR)  so  that  the 
baseline  system  test  configuration  can  be  presented  for 
major  review  at  the  same  time  as  the  final  detailed  test 
design. 


Figure  3.  Major  M&S  Features 
The  system  test  tool  is  developed  in  incremental 
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Figure  4.  Integrated  Ground  Test  Process 

UNCLASSIFIED 

3 


UNCLASSIFIED 


evolving  capability  of  the  system  test  tool  and  the 
embedded  system  test  article  representation.  This 
process  takes  test  objectives,  test  requirements,  and  data 
requirements  from  the  NMD  T&E  community  and  uses 
them  to  develop  a  coordinated  single  detailed  plan  for 
test  implementation  that  is  consistent  with  the  current 
maturity  level  of  the  system  test  configuration.  This 
design  is  currently  captured  for  use  by  the  test 
implementation  team  in  three  design  documents.  These 
documents  are  the  Detailed  Test  Plan  (DTP),  the 
Interface  Definition  Document  (IDD),  and  the  Data 
Management  Plan  (DMP),  Before  this  design  is 
implemented  it  must  pass  a  major  review  by  the  test 
conununity  at  the  TDR. 

Test  implementation  is  a  structured  process  that 
prepares  both  the  test  operations  team  and  the  test 
configuration  for  formal  runs-for-record  testing  under 
configuration  managed  conditions.  Test  data  is  then 
generated  during  runs-for-record  in  accordance  with  the 
detailed  design  for  designated  data  customers  that  will 
use  that  data  in  the  subsequent  data  analysis  activity.  A 
Quick  Look  report  is  prepared  soon  after  conclusion  of 
runs-for-record  to  provide  preliminary  test  execution 
information.  Test  data  is  then  reduced  and  delivered  to 
the  data  customers  as  specified  in  the  detailed  test 
design.  In  order  to  put  the  test  data  in  best  usable 
context,  a  Post  Test  Execution  Report  (PTER)  is 
prepared  that  describes  the  implementation  of  the  IGT 
along  with  the  processes  that  were  used  to  assess  the 
data  and  the  test  configuration  management  (CM).  The 
PTER  also  provides  tabular  and  summary 
documentation  of  any  hardware  and/or  software 
anomalies  encountered  during  runs-for-record.  This 
context  information  is  useful  to  the  data  analysts  for 
their  analyses. 

Data  analysis  by  IGT  data  customers  is 
independent  of  both  the  test  implementation  team  and 
the  system  test  tool  developer.  A  Post  Test  Analysis 
Report  (PTAR)  is  prepared  to  document  the  assessment 
of  results  of  an  IGT  as  related  to  the  test  objectives. 
Data  customers  are  invited  to  present  their  interim  or 
final  analyses  at  a  Data  and  Analysis  Review  as  the  last 
activity  in  the  IGT  process.  Results  of  these  analyses 
are  then  available  as  feedback  to  the  NMD  community 
to  aid  preparations  for  the  next  IGT. 


Value  Expected 

While  IFT  can  provide  but  a  limited  quantity  of 
highly  credible  data,  M&S  can  provide  large  amounts 
of  usable  data  if  M&S  can  be  made  creditable.  This 
places  high  value  on  a  workable  methodology  to 
transfer  IFT  credibility  to  M&S.  IGT  can  provide  this 
through  the  process  of  anchoring.  Anchoring  uses  an 
IGT  to  reproduce  the  high  quality  IFT  data  result  for  a 
given  scenario  or  point  of  interest.  Resolution  of 
differences  in  this  comparison  requires  improvements 
in  only  the  simulated  part  of  an  IGT,  which  is  minimal. 
The  rest  of  the  IGT  uses  the  same  hardware  and 
software  as  the  IFT  and  uses  accredited  test  community 
simulations  of  the  test  environment.  Point  validation  is 
considered  easier  for  IGT  than  M&S  for  this  reason. 
This  reproduction  of  results  provides  a  point  validation 
of  the  IGT  against  the  current  system  performance 
envelope.  Additional  IFT  can  provide  additional  point 
validations  at  different  points  on  the  performance 
envelope  and  can  give  the  IGT  high  credibility. 
Transfer  of  this  credibility  to  M&S  utilizes  data 
available  from  IGT  at  these  validation  points  as  well  as 
points  not  testable  in  IFT:  multiple  interceptors 
engaging  multiple  threat  objects  in  various  tactical 
geometries.  M&S  then  has  the  necessary  amounts  of 
high  quality  data  needed  to  perform  validation  of  all  the 
M&S  components.  M&S  can  produce  large  amounts  of 
credible  decision  data  to  include  statistical  data  to  be 
used  in  the  DRR. 

An  IGT  is  also  part  of  building  a  test  capability 
that  increases  in  value  for  each  subsequent  IGT.  Value 
is  increased  by  providing  confidence  in  the  reduction  of 
program  risk.  This  confidence  is  provided  by 
demonstrating  increasing  capability  in  the  T&E  of  the 
system  as  system  performance  increases  over  time. 
Aspects  of  this  capability  are  visible  as  1)  ability  to 
perform  system  assessments  against  system 
requirements,  2)  ability  to  document  complex  system 
tests,  3)  ability  to  train  personnel  to  integrate  a  system 
test  configuration,  conduct  a  test  and  reduce  data 
products,  4)  ability  to  perform  NMD  system  integration 
with  all  necessary  sub-components ^and  deal  effectively 
with  accreditation  requirements,  and  5)  establishing  test 
infrastructure  useful  for  long-term  IGT. 
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A  subtle  but  very  important  IGT  value  is 
associated  with  the  first  forced  integration  of  elements 
that  previously  have  not  been  required  to  operate 
together  in  system  context.  An  IGT  with  first  time 
element  participation  will  invariably  reveal  interactive 
issues  not  previously  anticipated.  Early  identification 
facilitates  resolution. 

Another  value  expected  is  the  establishment  of  a 
common  level  or  standard  of  test  configuration 
management  discipline.  Each  organizational  participant 
enters  the  IGT  process  with  differing  CM  standards. 
The  detailed  test  design  brings  these  standards  together 
at  the  test-tool-to-element  and  element-to-element 
interfaces.  This  same  discipline  is  used  to  maintain  an 
audit  trail  for  collected  test  data  credibility. 

Of  considerable  importance  is  the  value  of  forging 
a  common  set  of  test  expectations  from  the  test 
participants  who  come  from  different  test  backgrounds 
with  their  own  separate  semantics  and  practices.  This 
lack  of  common  test  expectations  is  most  evident 
during  the  detailed  test  planning  coordination  when  the 
proponents  of  the  top-down  requirements  approach 
clash  with  those  favoring  the  bottom-up  capability 
approach.  Both  approaches  have  merit.  The  system 
must  eventually  be  tested  against  NMD  system 
requirements  but  early  IGT  testing  must  use  available 
system  element  components  that  are  still  maturing  but 
currently  only  meet  pre-existing  long  lead  contract 
requirements.  Typically  the  system  engineer  would  take 
the  former  approach  and  the  elements  would  take  the 
latter  approach.  Both  approaches  must  be  balanced  in 
planning  an  IGT.  Whether  agreement  is  reached  on  all 
objectives  or  not,  a  common  understanding  emerges  on 
what  is  doable  in  a  given  IGT. 

n.  Detailed  Design  and  Implementation 
Detailed  Design 

Detailed  test  design  begins  with  initial  test  and 
data  requirements  analysis  and  early  research  into 
achievable  test  configuration  design.  A  Technical 
Interchange  Meeting  (TIM)  is  used  to  kick-off  formal 
detailed  test  design.  Participants  meet,  exchange 
information  and  begin  the  necessary  coordination.  This 


design  is  captured  in  detailed  design  documentation 
(DTP,  IDD,  DMP)  that  is  developed  and  reviewed 
incrementally.  The  NMD  test  community  influences 
this  design  through  information  input,  coordination, 
and  participation  in  major  reviews.  The  goal  is  to 
converge  the  detailed  design,  to  define  the  test 
interfaces  and  messages,  and  to  define  the  data 
management  process  for  identifying,  handling,  and 
distributing  specific  IGT  data  products.  The  purpose  of 
the  DTP  is  to  provide  details  for  the  Test  Director  to 
implement/execute  a  specific  IGT.  These  details 
include  1)  test  objectives,  2)  test  requirements 
(allocation  to  test  configuration),  3)  data  requirements 
(what,  how,  who  are  data  customers),  4)  scenarios,  5) 
implementation  activities  (pre-test,  test,  post-test),  6) 
test  configuration  description,  and  7)  test  management 
and  controls  (roles  and  responsibilities,  configuration 
management,  security  and  safety,  schedule  and  change 
process).  The  purpose  of  the  IDD  is  to  define  test 
interfaces  and  messages  for  use  by  the  Test  Director  to 
document  test  interfaces  for  interface  data  capability 
and  test  CM.  The  purpose  of  the  DMP  is  to  define  the 
data  management  process  for  identifying,  handling,  and 
distributing  IGT  data  products.  It  is  used  by  the  Test 
Director  to  document  the  data  management  process  and 
to  control  the  distribution  of  data  products.  Each 
design  increment  is  an  opportunity  to  resolve  design 
conflicts  and  update  the  detailed  design.  Design 
documentation  is  then  finalized  and  presented  for 
review  at  the  TDR. 

Implementation 

Test  implementation  begins  as  soon  as  possible 
after  an  authority  to  proceed  (ATP)  is  granted  at  the 
TDR.  This  ATP  is  generally  granted  at  the  TDR  upon 
evidence  that  the  test  configuration  and  the  detailed  test 
design  are  acceptable  and  both  have  been  placed  under 
test  configuration  management.  Test  configuration 
baseline  is  established  at  the  Test  Configuration 
Control  Board  (TCCB)  concurrent  with  the  TDR. 
Major  activities  in  test  implementation  are  1)  test 
preparation,  2)  runs-for-record,  and  3)  data 
reduction/reporting. 
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Test  Prenaration 

The  test  preparation  starts  with  turnover  of  the 
available  test  assets  baseline  from  the  system  test  tool 
developer  to  the  test  implementation  team  under  CM 
controlled  conditions.  This  involves  replacement  of  all 
system  test  tool  developer  computer/network  access 
privileges  with  test  CM  and  test  implementation  team 
access  privileges.  All  changes  to  the  baseline  test  assets 
and  detailed  test  design  that  are  found  necessary  for  test 
success  during  this  preparation  phase  are  made  and 
documented  for  a  test  CM  audit  trail.  Several  activities 
occur  in  parallel  during  preparation  to  save  test  time. 
Finalization  of  the  test  procedures  is  done  by  the 
implementing  operations  team  while  they  are 
performing  interface  verification  testing  to  verify  the 
baseline  test  configuration  as  advertised  in  the  baseline 
test  CM  documentation.  Concurrently,  test  setup  files 
for  each  scenario  to  be  run  are  generated  and  stored  for 
use  during  runs-for-record.  Also  concurrently,  post-run 
scripts  are  developed  for  use  in  data  reduction  to  meet 
the  specific  data  requirements  for  each  data  user 
identified  in  the  detailed  test  design.  When  all  setups, 
scripts  and  procedures  are  finalized  and  verified,  then 
the  implementing  operations  team  performs  dry  runs  in 
the  same  manner  as  planned  for  runs-for-record.  This 
involves  use  of  the  configuration  controlled  procedures 
and  the  test  configuration  by  the  actual  operations  crew 
to  collect  readiness  evidence  for  presentation  at  a  Test 
Readiness  Review  (TRR).  Authority  to  proceed  into  the 
formal  runs-for-record  is  granted  at  TRR  upon  evidence 
that  test  risk  is  acceptable. 

Run.s-for-Record 

Runs-for-record  are  formal  IGT  runs  witnessed  by 
test  observers  from  the  NMD  T&E  community  and 
Other  Government  Agencies  (OGAs).  They  are 
executed  under  the  direction  of  a  responsible  Test 
Director  with  direct  control  of  a  Test  Conductor  who 
manages  and  coordinates  the  day-to-day  conduct  of  the 
formal  runs.  The  test  operations  team  operates  the 
common  test  environment  (test  and  control,  global 
environment)  as  well  as  any  necessary  NTE  equipment. 
Each  element  node  has  a  designated  PTE  operator(s). 
Facility  support  personnel  and  a  system  test  tool 
technical  expert  are  on-call  as  needed  by  the  Test 


Conductor.  All  operational  participants  are  tasked  by 
the  Test  Conductor  to  execute  the  runs. 

Data  Reduction  /  Reporting 

After  each  test  run,  raw  captured  data  is 
transferred  from  the  several  collection  points  to  a  single 
workstation  for  test  CM  identification  and  control. 
Between  real  time  test  runs,  the  data  is  available  for 
spot  check  by  the  test  operators  for  null,  incomplete,  or 
corrupt  data  sets  due  to  vagaries  of  the  test  such  as 
electrical  storm  induced  errors.  Spot  checks  only  are 
available  since  the  data  is  not  yet  in  human  readable 
form  and  the  amount  of  data  is  very  large.  However,  if 
a  spot  check  reveals  a  problem  at  this  point,  the  Test 
Conductor  has  the  option  to  repeat  the  test  run.  This 
option  provides  a  method  for  confirming  that  requested 
data  for  designated  data  customers  has  actually  been 
collected  before  releasing  the  system  test  tool.  During 
post-test  data  reduction,  a  post  processor  converts  the 
data  to  ASCII  format  and  separates  it  into  files 
according  to  data  requests  documented  in  the  DMP. 
The  resulting  processed  data  files  are  also  brought 
under  test  CM.  At  this  point  both  raw  and  processed 
data  files  are  available  for  transfer  to  removable  media 
for  delivery  to  DMP-specified  data  customers. 

Daily  reporting  by  the  Test  Conductor  to  the  Test 
Director  is  provided  for  management  visibility  and  to 
provide  an  opportunity  to  identify  any  daily  run 
problems  requiring  attention  before  start  of  the  next 
day’s  runs.  A  Quick  Look  Report  is  prepared  within  48 
hours  of  runs-for-record  completion.  This  report  is 
coordinated  with  all  on-site  operator  participants  and 
identifies  whether  all  data  necessary  to  meet  test 
objectives  was  collected  as  specified  in  the  DTP.  It  also 
contains  preliminary  assessments  of  the  completed  test 
runs.  A  PTER  is  prepared  not  only  to  document  the  test 
execution  but  to  aid  data  customers  in  appropriate  use 
of  data.  This  report  describes  the  test  as  performed, 
provides  a  test  execution  and  data  collection 
assessment,  reports  anomalies,  identifies  corrective 
actions  required,  provides  lessons  learned,  and  gives 
reconunendations  for  future  IGT.  Inputs,  reports,  and 
conclusions  are  solicited  from  on-site  test  participants 
for  inclusion. 
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Recently  Identified  Value 

Important  value  is  placed  on  the  ability  to  shorten 
the  data  reduction/analysis/lessons  learned  timeline. 
This  ability  will  provide  timely  information  to  allow 
influence  of  the  design  of  the  following  IGTs  in  a  very 
compressed  and  over-lapping  test  schedule.  Achieving 
this  ability  remains  an  IGT  challenge  due  to  issues 
associated  with  the  amount  of  generated  data,  the  need 
for  credible  test  CM,  the  speed  of  data  media  copying 
equipment,  and  security  requirements.  The  magnitude 
of  just  the  data  amount  issue  is  growing  with  each 
capability  improvement  made  to  the  system  test  tool  as 
shown  in  Figure  5.  This  figure  provides  data  collection 
totals  for  the  recent  IGT  lA  conducted  in  April  and 
May  of  this  year  and  compares  them  with  totals 
observed  for  the  Integrated  Test  3  (ITS)  which  was  an 
informal  IGT-like  ground  test  conducted  in  the  summer 
of  1997.  ITS  involved  additional  runs  for  equal 
treatment  of  competing  Exo-atmospheric  Kill  Vehicle 
(EKV)  contractor  test  articles  that  participated. 
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Figure  5.  Demonstrated  Data  Collection 


test  configuration.  Much  work  overlaps  due  to 
compressed  scheduling.  The  solution  is  to  minimize 
test  run  cycle  time.  Run  cycle  time  is  the  sum  total  of 
the  contiguous  clock  time  required  to  setup  a  specific 
run,  to  execute  that  real  time  run  and  to  move  collected 
run  data  to  a  safe  CM  location  while  initializing  for  the 
next  cycle.  Observed  average  cycle  times  are  shown  in 
Figure  6  for  the  previously  mentioned  ITS  and  IGT  lA 
tests.  Note  that  IGT  lA  cycle  time  was  longer  than  that 
for  ns.  This  is  attributed  to  the  addition  of  a  complex 
node  to  the  test  configuration.  Recycle  time  is  expected 
to  grow  as  further  additional  nodes  are  added  to  the  test 
configurations  for  future  IGT.  Also  note  that  average 
cycle  times  for  both  ITS  and  IGT  lA  were  decreased 
during  runs-for-record  compared  to  the  preceding 
averages  observed  during  the  test  preparation  phase  of 
test  implementation.  This  valuable  decrease  is 
attributed  to  test  operator  practice  with  a  stable 
configuration  managed  test  configuration  and  test 
procedures. 


Prep  Record 
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Figure  6.  Demonstrated  Cycle  Time 


Variations 


Element  organizations  place  a  high  value  on  the 
control  and  management  of  data  and  information  about 
their  element  performance.  This  reflects  a  legitimate 
point  of  view  for  element  organizations  but  it  remains 
an  issue  that  needs  to  be  balanced  against  the  needs  of 
the  NMD  system.  Resolution  of  this  issue  is  expected  as 
elements  are  brought  under  the  management  of  the 
NMD  Lead  System  Integrator  (LSI). 

The  ability  to  maximize  the  number  of  runs 
achievable  per  work  shift  is  valuable  in  scheduling 
scarce  dedicated  system  test  tool  and  facility  time.  It 
must  be  noted  that  IGT  implementation  schedules 
compete  with  time  needed  for  developing  the  next  IGT 


There  are  several  possible  variations  of  the  IGT 
process.  These  differ  from  IGT  in  degree  of  formality, 
visibility  to  the  test  community,  degree  of  tested 
functionality,  purpose,  level  of  configuration 
management,  OGA  participation,  and  availability  of 
generated  data.  Examples  of  these  are  System 
Development  Test  (SDT),  Integrated  Element  Test 
(lET),  Integrated  Test  (IT),  System  Interface  Test 
(SIT),  Readiness  Test  Assessment  (RTA),  IFT  Pre¬ 
mission  Test  (PMT)  and  Element  Test  (ET).  Additional 
variations  are  possible. 
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An  SDT  might  focus  only  on  whether  the  system 
test  tool  functionality  meets  development  requirements 
with  an  embedded  system.  An  lET,  FT,  or  SIT  might 
have  IGT-like  test  configurations  but  be  conducted 
informally  with  narrow  objectives,  reduced  test  CM, 
and  limited  availability  of  the  data  products.  An  RTA 
might  be  conducted  informally  with  reduced  test  CM 
and  a  preliminary  IGT  test  configuration  for  purpose  of 
maintaining  and  increasing  IGT  operator  skills.  A  PMT 
would  be  conducted  to  reduce  risk  for  a  subsequent  IFT 
with  similar  NMD  system  representation.  Finally,  an 
ET  might  be  conducted  for  a  specific  element  in  a 
system  context  but  involve  both  limited  visibility  and 
limited  data  availability  under  element  control. 

in.  Lessons  Learned 

Lessons  learned  to  date  from  the  recent  IGT  lA 
can  be  grouped  into  categories  associated  with  1)  test 
design,  2)  procedures  and  3)  hardware  and  software. 

Test  Design 

Early  element  and  system  engineer  involvement  is 
needed  to  define  test  objectives,  test  requirements,  and 
the  test  configuration.  It  was  observed  that  there  is  a 
tendency  of  the  T&E  test  community  to  delay 
identification  of  issues  critical  to  the  final  detailed 
design.  It  was  also  observed  that  key  issues  surfaced 
fastest  when  system  engineer  and  element 
representatives  participated  in  face-to-face  coordination 
as  part  of  the  detailed  design.  It  will  be  valuable  to 
future  IGT  for  small  design  working  group  sessions  to 
be  used  early  in  the  detailed  design  as  the  forums  for 
the  necessary  system  engineer  and  element 
involvement. 

The  level  of  element  test  configuration 
management  needs  to  be  more  uniform  across  all 
elements  with  standards  established  as  part  of  the 
detailed  design.  Ability  to  accurately  document  the  test 
configuration  and  interfaces  will  be  significantly 
improved  by  this  measure. 

Data  customers  need  to  provide  more  complete 
addresses  in  the  DMP  of  Points  of  Contact  (POC)  and 
alternates  that  are  authorized  to  receive  the  data.  Large 


amounts  of  classified  data  are  involved  and  needed  for 
timely  analysis.  This  measure  will  avoid  delays  in  data 
delivery  and  subsequent  start  of  analysis.  It  will  also 
provide  a  means  for  dealing  with  absent  POCs  and 
POCs  that  change  subsequent  to  the  DMP  finalization. 

There  is  a  need  for  an  improved  method  to  obtain 
test  community  input  and  coordination  for  the  detailed 
design  documentation.  IGT  can  borrow  the  effective 
technique  used  by  IFT  of  a  page-by-page  and  sit-down 
review  of  detailed  design  documentation. 

Early  identification  of  collected  data  formats  is 
necessary  to  aid  data  analysis  planning.  Advance 
knowledge  in  this  area  will  allow  data  customers  to 
prepare  and  test  advance  analysis  templates  that  will 
speed  data  analysis  upon  data  delivery.  This  measure 
will  involve  more  accurate  early  data  format 
identification  by  both  the  elements  and  the  system  test 
tool  developer  for  inclusion  in  the  IDD. 

Element  data  collection  capability  needs  to  be 
identified  in  the  detailed  design  documentation  and 
implemented  in  the  test  configuration  prior  to  the  TAA 
milestone.  This  measure  will  allow  data  customers  to 
know  in  advance  of  test  implementation  what  element 
data  will  be  available  to  support  their  analyses. 

The  accreditation  process  for  elements  in  the  test 
configuration  needs  clarification  in  areas  of  required 
tests,  data,  schedule,  primary  POCs,  and  any  other 
activities  required.  Element  representatives  were 
unaware  of  precise  requirements  for  each  accreditation 
phase  in  IGT  1  A. 

Early  development  of  accreditation  acceptability 
criteria  is  needed  with  clear  decision  authority 
identified.  It  will  be  valuable  to  the  next  IGT  to  develop 
acceptability  criteria  early  with  inputs  from 
representatives  of  elements  and  Operational  Test 
Agencies  (OTAs). 

Procedures 

Training  exercises  are  required  to  train  operators 
and  verify  procedures  before  TRR.  This  measure  is 
critical  to  IGT  effectiveness  in  a  compressed  schedule. 
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Test  operations  team  capability  needs  to  be 
maintained  by  conducting  RTAs  on  a  regular  basis. 
Operations  team  must  be  skilled  on  latest  modifications 
to  system  test  tool  and  element  functionality. 

It  is  important  to  formally  document  all  IGT 
problems  encountered  and  track  them  until  resolved. 
Otherwise,  it  is  probable  that  the  same  problem  will  be 
encountered  on  a  future  IGT. 

An  independent  procedure  is  needed  to  verify  the 
proper  test  configuration  file  has  been  established 
without  configuration  setup  error.  This  lesson  was 
learned  and  implemented  in  IGT  1  A. 

Element  personnel  need  direct  access  to  test  data 
and  reports  fi’om  each  system  test  tool/element 
integration  phase.  Data  and  reports  were  not  available 
until  all  integration  test  phases  were  complete  during 
IGT  lA  system  test  tool  development  and  element  test 
article  integration. 

A  radar  node  procedure  modification  is  needed  to 
ensure  the  proper  NTE  clock  time  is  used  by  the  radar 
element  node  operator.  This  lesson  was  learned  and 
implemented  in  IGT  lA. 

A  radar  node  procedure  modification  is  needed  to 
ensure  both  PTE  processor  machines  are  operating  at 
proper  clock  frequency  before  starting  test.  This  lesson 
was  also  learned  and  implemented  during  IGT  lA. 

Hardware  and  Software 

Software  script  for  fetch  of  BMC3  post-run 
element  data  needs  error  handling  or  flagging  to  reduce 
potential  for  operator  error. 

System  test  tool  timeline  software  needs 
modification  to  allow  playback  of  collected  data  after 
CM  data  lock-down. 

The  system  test  tool  needs  modification  to  provide 
faster  data  archiving  of  large  amounts  of  collected  data. 
The  current  tape  archiving  system  is  unacceptably  slow. 

A  system  test  tool  software  modification  and 
improved  software  documentation  are  needed  to 


reduce  risk  of  improper  setting  of  data  collection 
latches  during  pre-mission  test  configuration  file  setup. 
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